{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Redcard Exploratory Data Analysis\n",
    "\n",
    "This dataset is taken from a fantastic paper that looks to see how analytical choices made by different data science teams on the same dataset in an attempt to answer the same research question affect the final outcome.\n",
    "\n",
    "[Many analysts, one dataset: Making transparent how variations in analytical choices affect results](https://osf.io/gvm2z/)\n",
    "\n",
    "The data can be found [here](https://osf.io/47tnc/).\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## The Task\n",
    "\n",
    "Do an Exploratory Data Analysis on the redcard dataset. Keeping in mind the question is the following: **Are soccer referees more likely to give red cards to dark-skin-toned players than light-skin-toned players?**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fetching package metadata ...........\n",
      "Solving package specifications: .\n",
      "\n",
      "Package plan for installation in environment /home/nbcommon/anaconda3_410:\n",
      "\n",
      "The following NEW packages will be INSTALLED:\n",
      "\n",
      "    pandas-profiling: 1.3.0-py35_0  conda-forge\n",
      "\n",
      "The following packages will be SUPERSEDED by a higher-priority channel:\n",
      "\n",
      "    conda:            4.3.17-py35_0             --> 4.2.13-py35_0 conda-forge\n",
      "    conda-env:        2.6.0-0                   --> 2.6.0-0       conda-forge\n",
      "\n",
      "conda-env-2.6. 100% |################################| Time: 0:00:00   1.17 MB/s\n",
      "conda-4.2.13-p 100% |################################| Time: 0:00:00   1.44 MB/s\n",
      "pandas-profili 100% |################################| Time: 0:00:00 663.79 kB/s\n",
      "Collecting missingno\n",
      "  Downloading missingno-0.3.5.tar.gz\n",
      "Requirement already satisfied (use --upgrade to upgrade): numpy in /home/nbcommon/anaconda3_410/lib/python3.5/site-packages (from missingno)\n",
      "Requirement already satisfied (use --upgrade to upgrade): matplotlib in /home/nbcommon/anaconda3_410/lib/python3.5/site-packages (from missingno)\n",
      "Requirement already satisfied (use --upgrade to upgrade): scipy in /home/nbcommon/anaconda3_410/lib/python3.5/site-packages (from missingno)\n",
      "Requirement already satisfied (use --upgrade to upgrade): seaborn in /home/nbcommon/anaconda3_410/lib/python3.5/site-packages (from missingno)\n",
      "Requirement already satisfied (use --upgrade to upgrade): python-dateutil in /home/nbcommon/anaconda3_410/lib/python3.5/site-packages (from matplotlib->missingno)\n",
      "Requirement already satisfied (use --upgrade to upgrade): pytz in /home/nbcommon/anaconda3_410/lib/python3.5/site-packages (from matplotlib->missingno)\n",
      "Requirement already satisfied (use --upgrade to upgrade): cycler in /home/nbcommon/anaconda3_410/lib/python3.5/site-packages (from matplotlib->missingno)\n",
      "Requirement already satisfied (use --upgrade to upgrade): pyparsing!=2.0.4,>=1.5.6 in /home/nbcommon/anaconda3_410/lib/python3.5/site-packages (from matplotlib->missingno)\n",
      "Requirement already satisfied (use --upgrade to upgrade): six>=1.5 in /home/nbcommon/anaconda3_410/lib/python3.5/site-packages (from python-dateutil->matplotlib->missingno)\n",
      "Building wheels for collected packages: missingno\n",
      "  Running setup.py bdist_wheel for missingno ... \u001b[?25l-\b \b\\\b \bdone\n",
      "\u001b[?25h  Stored in directory: /home/nbuser/.cache/pip/wheels/f3/9f/31/38d2fad2bd1ac3ac70a2d159c61515de5825296429a4f13056\n",
      "Successfully built missingno\n",
      "Installing collected packages: missingno\n",
      "Successfully installed missingno-0.3.5\n",
      "\u001b[33mYou are using pip version 8.1.2, however version 9.0.1 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!conda install -c conda-forge pandas-profiling -y \n",
    "\n",
    "!pip install missingno"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/nbuser/anaconda3_410/lib/python3.5/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.\n",
      "  warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')\n",
      "/home/nbuser/anaconda3_410/lib/python3.5/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.\n",
      "  warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format='retina'\n",
    "\n",
    "from __future__ import absolute_import, division, print_function\n",
    "import matplotlib as mpl\n",
    "from matplotlib import pyplot as plt\n",
    "from matplotlib.pyplot import GridSpec\n",
    "import seaborn as sns\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import os, sys\n",
    "from tqdm import tqdm\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "sns.set_context(\"poster\", font_scale=1.3)\n",
    "\n",
    "import missingno as msno\n",
    "import pandas_profiling\n",
    "\n",
    "from sklearn.datasets import make_blobs\n",
    "import time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## About the Data\n",
    "\n",
    "> The dataset is available as a list with 146,028 dyads of players and referees and includes details from players, details from referees and details regarding the interactions of player-referees. A summary of the variables of interest can be seen below. A detailed description of all variables included can be seen in the README file on the project website. \n",
    "\n",
    "> From a company for sports statistics, we obtained data and profile photos from all soccer players (N = 2,053) playing in the first male divisions of England, Germany, France and Spain in the 2012-2013 season and all referees (N = 3,147) that these players played under in their professional career (see Figure 1). We created a dataset of playerâreferee dyads including the number of matches players and referees encountered each other and our dependent variable, the number of red cards given to a player by a particular referee throughout all matches the two encountered each other.\n",
    "\n",
    "> -- https://docs.google.com/document/d/1uCF5wmbcL90qvrk_J27fWAvDcDNrO9o_APkicwRkOKc/edit\n",
    "\n",
    "\n",
    "| Variable Name: | Variable Description: | \n",
    "| -- | -- | \n",
    "| playerShort | short player ID | \n",
    "| player | player name | \n",
    "| club | player club | \n",
    "| leagueCountry | country of player club (England, Germany, France, and Spain) | \n",
    "| height | player height (in cm) | \n",
    "| weight | player weight (in kg) | \n",
    "| position | player position | \n",
    "| games | number of games in the player-referee dyad | \n",
    "| goals | number of goals in the player-referee dyad | \n",
    "| yellowCards | number of yellow cards player received from the referee | \n",
    "| yellowReds | number of yellow-red cards player received from the referee | \n",
    "| redCards | number of red cards player received from the referee | \n",
    "| photoID | ID of player photo (if available) | \n",
    "| rater1 | skin rating of photo by rater 1 | \n",
    "| rater2 | skin rating of photo by rater 2 | \n",
    "| refNum | unique referee ID number (referee name removed for anonymizing purposes) | \n",
    "| refCountry | unique referee country ID number | \n",
    "| meanIAT | mean implicit bias score (using the race IAT) for referee country | \n",
    "| nIAT | sample size for race IAT in that particular country | \n",
    "| seIAT | standard error for mean estimate of race IAT   | \n",
    "| meanExp | mean explicit bias score (using a racial thermometer task) for referee country | \n",
    "| nExp | sample size for explicit bias in that particular country | \n",
    "| seExp |  standard error for mean estimate of explicit bias measure | \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What the teams found\n",
    "\n",
    "\n",
    "### Choices in model features\n",
    "\n",
    "The following is the covariates chosen for the respective models: \n",
    "\n",
    "<img src=\"figures/covariates.png\" width=80%;>\n",
    "\n",
    "\n",
    "### Choices in modeling\n",
    "\n",
    "Of the many choices made by the team, here is a small selection of the models used to answer this question:\n",
    "\n",
    "\n",
    "<img src=\"figures/models.png\" width=80%;>\n",
    "\n",
    "\n",
    "## Final Results\n",
    "\n",
    " - 0 teams: negative effect\n",
    " - 9 teams: no significant relationship\n",
    " - 20 teams: finding a positive effect\n",
    "\n",
    "<img src=\"figures/results.png\" width=80%;>\n",
    "\n",
    "Above image from: http://fivethirtyeight.com/features/science-isnt-broken/#part2\n",
    "\n",
    "\n",
    "> â¦selecting randomly from the present teams, there would have been a 69% probability of reporting a positive result and a 31% probability of reporting a null effect. This raises the possibility that many research projects contain hidden uncertainty due to the wide range of analytic choices available to the researchers. -- Silberzahn, R., Uhlmann, E. L., Martin, D. P., Pasquale, Aust, F., Awtrey, E. C., â¦ Nosek, B. A. (2015, August 20). Many analysts, one dataset: Making transparent how variations in analytical choices affect results. Retrieved from osf.io/gvm2z\n",
    "\n",
    "\n",
    "Images and data from: Silberzahn, R., Uhlmann, E. L., Martin, D. P., Pasquale, Aust, F., Awtrey, E. C., â¦ Nosek, B. A. (2015, August 20). Many analysts, one dataset: Making transparent how variations in analytical choices affect results. Retrieved from osf.io/gvm2z"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## General tips\n",
    "\n",
    "- Before plotting/joining/doing something, have a question or hypothesis that you want to investigate\n",
    "- Draw a plot of what you want to see on paper to sketch the idea\n",
    "- Write it down, then make the plan on how to get there\n",
    "- How do you know you aren't fooling yourself\n",
    "- What else can I check if this is actually true?\n",
    "- What evidence could there be that it's wrong?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Uncomment one of the following lines and run the cell:\n",
    "\n",
    "# df = pd.read_csv(\"../data/redcard/redcard.csv.gz\", compression='gzip')\n",
    "df = pd.read_csv(\"https://github.com/cmawer/pycon-2017-eda-tutorial/raw/master/data/redcard/redcard.csv.gz\",\n",
    "                 compression='gzip')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(146028, 28)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>playerShort</th>\n",
       "      <th>player</th>\n",
       "      <th>club</th>\n",
       "      <th>leagueCountry</th>\n",
       "      <th>birthday</th>\n",
       "      <th>height</th>\n",
       "      <th>weight</th>\n",
       "      <th>position</th>\n",
       "      <th>games</th>\n",
       "      <th>victories</th>\n",
       "      <th>...</th>\n",
       "      <th>rater2</th>\n",
       "      <th>refNum</th>\n",
       "      <th>refCountry</th>\n",
       "      <th>Alpha_3</th>\n",
       "      <th>meanIAT</th>\n",
       "      <th>nIAT</th>\n",
       "      <th>seIAT</th>\n",
       "      <th>meanExp</th>\n",
       "      <th>nExp</th>\n",
       "      <th>seExp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>lucas-wilchez</td>\n",
       "      <td>Lucas Wilchez</td>\n",
       "      <td>Real Zaragoza</td>\n",
       "      <td>Spain</td>\n",
       "      <td>31.08.1983</td>\n",
       "      <td>177.0</td>\n",
       "      <td>72.0</td>\n",
       "      <td>Attacking Midfielder</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.50</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>GRC</td>\n",
       "      <td>0.326391</td>\n",
       "      <td>712.0</td>\n",
       "      <td>0.000564</td>\n",
       "      <td>0.396000</td>\n",
       "      <td>750.0</td>\n",
       "      <td>0.002696</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>john-utaka</td>\n",
       "      <td>John Utaka</td>\n",
       "      <td>Montpellier HSC</td>\n",
       "      <td>France</td>\n",
       "      <td>08.01.1982</td>\n",
       "      <td>179.0</td>\n",
       "      <td>82.0</td>\n",
       "      <td>Right Winger</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.75</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>ZMB</td>\n",
       "      <td>0.203375</td>\n",
       "      <td>40.0</td>\n",
       "      <td>0.010875</td>\n",
       "      <td>-0.204082</td>\n",
       "      <td>49.0</td>\n",
       "      <td>0.061504</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>abdon-prats</td>\n",
       "      <td>AbdÃ³n Prats</td>\n",
       "      <td>RCD Mallorca</td>\n",
       "      <td>Spain</td>\n",
       "      <td>17.12.1992</td>\n",
       "      <td>181.0</td>\n",
       "      <td>79.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>ESP</td>\n",
       "      <td>0.369894</td>\n",
       "      <td>1785.0</td>\n",
       "      <td>0.000229</td>\n",
       "      <td>0.588297</td>\n",
       "      <td>1897.0</td>\n",
       "      <td>0.001002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>pablo-mari</td>\n",
       "      <td>Pablo MarÃ­</td>\n",
       "      <td>RCD Mallorca</td>\n",
       "      <td>Spain</td>\n",
       "      <td>31.08.1993</td>\n",
       "      <td>191.0</td>\n",
       "      <td>87.0</td>\n",
       "      <td>Center Back</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>ESP</td>\n",
       "      <td>0.369894</td>\n",
       "      <td>1785.0</td>\n",
       "      <td>0.000229</td>\n",
       "      <td>0.588297</td>\n",
       "      <td>1897.0</td>\n",
       "      <td>0.001002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ruben-pena</td>\n",
       "      <td>RubÃ©n PeÃ±a</td>\n",
       "      <td>Real Valladolid</td>\n",
       "      <td>Spain</td>\n",
       "      <td>18.07.1991</td>\n",
       "      <td>172.0</td>\n",
       "      <td>70.0</td>\n",
       "      <td>Right Midfielder</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>ESP</td>\n",
       "      <td>0.369894</td>\n",
       "      <td>1785.0</td>\n",
       "      <td>0.000229</td>\n",
       "      <td>0.588297</td>\n",
       "      <td>1897.0</td>\n",
       "      <td>0.001002</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows Ã 28 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     playerShort         player             club leagueCountry    birthday  \\\n",
       "0  lucas-wilchez  Lucas Wilchez    Real Zaragoza         Spain  31.08.1983   \n",
       "1     john-utaka     John Utaka  Montpellier HSC        France  08.01.1982   \n",
       "2    abdon-prats    AbdÃ³n Prats     RCD Mallorca         Spain  17.12.1992   \n",
       "3     pablo-mari     Pablo MarÃ­     RCD Mallorca         Spain  31.08.1993   \n",
       "4     ruben-pena     RubÃ©n PeÃ±a  Real Valladolid         Spain  18.07.1991   \n",
       "\n",
       "   height  weight              position  games  victories    ...     rater2  \\\n",
       "0   177.0    72.0  Attacking Midfielder      1          0    ...       0.50   \n",
       "1   179.0    82.0          Right Winger      1          0    ...       0.75   \n",
       "2   181.0    79.0                   NaN      1          0    ...        NaN   \n",
       "3   191.0    87.0           Center Back      1          1    ...        NaN   \n",
       "4   172.0    70.0      Right Midfielder      1          1    ...        NaN   \n",
       "\n",
       "   refNum  refCountry  Alpha_3   meanIAT    nIAT     seIAT   meanExp    nExp  \\\n",
       "0       1           1      GRC  0.326391   712.0  0.000564  0.396000   750.0   \n",
       "1       2           2      ZMB  0.203375    40.0  0.010875 -0.204082    49.0   \n",
       "2       3           3      ESP  0.369894  1785.0  0.000229  0.588297  1897.0   \n",
       "3       3           3      ESP  0.369894  1785.0  0.000229  0.588297  1897.0   \n",
       "4       3           3      ESP  0.369894  1785.0  0.000229  0.588297  1897.0   \n",
       "\n",
       "      seExp  \n",
       "0  0.002696  \n",
       "1  0.061504  \n",
       "2  0.001002  \n",
       "3  0.001002  \n",
       "4  0.001002  \n",
       "\n",
       "[5 rows x 28 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>min</th>\n",
       "      <th>25%</th>\n",
       "      <th>50%</th>\n",
       "      <th>75%</th>\n",
       "      <th>max</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>height</th>\n",
       "      <td>145765.0</td>\n",
       "      <td>181.935938</td>\n",
       "      <td>6.738726</td>\n",
       "      <td>1.610000e+02</td>\n",
       "      <td>177.000000</td>\n",
       "      <td>182.000000</td>\n",
       "      <td>187.000000</td>\n",
       "      <td>2.030000e+02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>weight</th>\n",
       "      <td>143785.0</td>\n",
       "      <td>76.075662</td>\n",
       "      <td>7.140906</td>\n",
       "      <td>5.400000e+01</td>\n",
       "      <td>71.000000</td>\n",
       "      <td>76.000000</td>\n",
       "      <td>81.000000</td>\n",
       "      <td>1.000000e+02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>games</th>\n",
       "      <td>146028.0</td>\n",
       "      <td>2.921166</td>\n",
       "      <td>3.413633</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>4.700000e+01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>victories</th>\n",
       "      <td>146028.0</td>\n",
       "      <td>1.278344</td>\n",
       "      <td>1.790725</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.900000e+01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ties</th>\n",
       "      <td>146028.0</td>\n",
       "      <td>0.708241</td>\n",
       "      <td>1.116793</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.400000e+01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>defeats</th>\n",
       "      <td>146028.0</td>\n",
       "      <td>0.934581</td>\n",
       "      <td>1.383059</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.800000e+01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>goals</th>\n",
       "      <td>146028.0</td>\n",
       "      <td>0.338058</td>\n",
       "      <td>0.906481</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.300000e+01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>yellowCards</th>\n",
       "      <td>146028.0</td>\n",
       "      <td>0.385364</td>\n",
       "      <td>0.795333</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.400000e+01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>yellowReds</th>\n",
       "      <td>146028.0</td>\n",
       "      <td>0.011381</td>\n",
       "      <td>0.107931</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>redCards</th>\n",
       "      <td>146028.0</td>\n",
       "      <td>0.012559</td>\n",
       "      <td>0.112889</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rater1</th>\n",
       "      <td>124621.0</td>\n",
       "      <td>0.264255</td>\n",
       "      <td>0.295382</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>1.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rater2</th>\n",
       "      <td>124621.0</td>\n",
       "      <td>0.302862</td>\n",
       "      <td>0.293020</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>1.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>refNum</th>\n",
       "      <td>146028.0</td>\n",
       "      <td>1534.827444</td>\n",
       "      <td>918.736625</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>641.000000</td>\n",
       "      <td>1604.000000</td>\n",
       "      <td>2345.000000</td>\n",
       "      <td>3.147000e+03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>refCountry</th>\n",
       "      <td>146028.0</td>\n",
       "      <td>29.642842</td>\n",
       "      <td>27.496189</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>7.000000</td>\n",
       "      <td>21.000000</td>\n",
       "      <td>44.000000</td>\n",
       "      <td>1.610000e+02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>meanIAT</th>\n",
       "      <td>145865.0</td>\n",
       "      <td>0.346276</td>\n",
       "      <td>0.032246</td>\n",
       "      <td>-4.725423e-02</td>\n",
       "      <td>0.334684</td>\n",
       "      <td>0.336628</td>\n",
       "      <td>0.369894</td>\n",
       "      <td>5.737933e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>nIAT</th>\n",
       "      <td>145865.0</td>\n",
       "      <td>19697.411216</td>\n",
       "      <td>127126.197143</td>\n",
       "      <td>2.000000e+00</td>\n",
       "      <td>1785.000000</td>\n",
       "      <td>2882.000000</td>\n",
       "      <td>7749.000000</td>\n",
       "      <td>1.975803e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>seIAT</th>\n",
       "      <td>145865.0</td>\n",
       "      <td>0.000631</td>\n",
       "      <td>0.004736</td>\n",
       "      <td>2.235373e-07</td>\n",
       "      <td>0.000055</td>\n",
       "      <td>0.000151</td>\n",
       "      <td>0.000229</td>\n",
       "      <td>2.862871e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>meanExp</th>\n",
       "      <td>145865.0</td>\n",
       "      <td>0.452026</td>\n",
       "      <td>0.217469</td>\n",
       "      <td>-1.375000e+00</td>\n",
       "      <td>0.336101</td>\n",
       "      <td>0.356446</td>\n",
       "      <td>0.588297</td>\n",
       "      <td>1.800000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>nExp</th>\n",
       "      <td>145865.0</td>\n",
       "      <td>20440.233860</td>\n",
       "      <td>130615.745103</td>\n",
       "      <td>2.000000e+00</td>\n",
       "      <td>1897.000000</td>\n",
       "      <td>3011.000000</td>\n",
       "      <td>7974.000000</td>\n",
       "      <td>2.029548e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>seExp</th>\n",
       "      <td>145865.0</td>\n",
       "      <td>0.002994</td>\n",
       "      <td>0.019723</td>\n",
       "      <td>1.043334e-06</td>\n",
       "      <td>0.000225</td>\n",
       "      <td>0.000586</td>\n",
       "      <td>0.001002</td>\n",
       "      <td>1.060660e+00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                count          mean            std           min          25%  \\\n",
       "height       145765.0    181.935938       6.738726  1.610000e+02   177.000000   \n",
       "weight       143785.0     76.075662       7.140906  5.400000e+01    71.000000   \n",
       "games        146028.0      2.921166       3.413633  1.000000e+00     1.000000   \n",
       "victories    146028.0      1.278344       1.790725  0.000000e+00     0.000000   \n",
       "ties         146028.0      0.708241       1.116793  0.000000e+00     0.000000   \n",
       "defeats      146028.0      0.934581       1.383059  0.000000e+00     0.000000   \n",
       "goals        146028.0      0.338058       0.906481  0.000000e+00     0.000000   \n",
       "yellowCards  146028.0      0.385364       0.795333  0.000000e+00     0.000000   \n",
       "yellowReds   146028.0      0.011381       0.107931  0.000000e+00     0.000000   \n",
       "redCards     146028.0      0.012559       0.112889  0.000000e+00     0.000000   \n",
       "rater1       124621.0      0.264255       0.295382  0.000000e+00     0.000000   \n",
       "rater2       124621.0      0.302862       0.293020  0.000000e+00     0.000000   \n",
       "refNum       146028.0   1534.827444     918.736625  1.000000e+00   641.000000   \n",
       "refCountry   146028.0     29.642842      27.496189  1.000000e+00     7.000000   \n",
       "meanIAT      145865.0      0.346276       0.032246 -4.725423e-02     0.334684   \n",
       "nIAT         145865.0  19697.411216  127126.197143  2.000000e+00  1785.000000   \n",
       "seIAT        145865.0      0.000631       0.004736  2.235373e-07     0.000055   \n",
       "meanExp      145865.0      0.452026       0.217469 -1.375000e+00     0.336101   \n",
       "nExp         145865.0  20440.233860  130615.745103  2.000000e+00  1897.000000   \n",
       "seExp        145865.0      0.002994       0.019723  1.043334e-06     0.000225   \n",
       "\n",
       "                     50%          75%           max  \n",
       "height        182.000000   187.000000  2.030000e+02  \n",
       "weight         76.000000    81.000000  1.000000e+02  \n",
       "games           2.000000     3.000000  4.700000e+01  \n",
       "victories       1.000000     2.000000  2.900000e+01  \n",
       "ties            0.000000     1.000000  1.400000e+01  \n",
       "defeats         1.000000     1.000000  1.800000e+01  \n",
       "goals           0.000000     0.000000  2.300000e+01  \n",
       "yellowCards     0.000000     1.000000  1.400000e+01  \n",
       "yellowReds      0.000000     0.000000  3.000000e+00  \n",
       "redCards        0.000000     0.000000  2.000000e+00  \n",
       "rater1          0.250000     0.250000  1.000000e+00  \n",
       "rater2          0.250000     0.500000  1.000000e+00  \n",
       "refNum       1604.000000  2345.000000  3.147000e+03  \n",
       "refCountry     21.000000    44.000000  1.610000e+02  \n",
       "meanIAT         0.336628     0.369894  5.737933e-01  \n",
       "nIAT         2882.000000  7749.000000  1.975803e+06  \n",
       "seIAT           0.000151     0.000229  2.862871e-01  \n",
       "meanExp         0.356446     0.588297  1.800000e+00  \n",
       "nExp         3011.000000  7974.000000  2.029548e+06  \n",
       "seExp           0.000586     0.001002  1.060660e+00  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.describe().T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "playerShort       object\n",
       "player            object\n",
       "club              object\n",
       "leagueCountry     object\n",
       "birthday          object\n",
       "height           float64\n",
       "weight           float64\n",
       "position          object\n",
       "games              int64\n",
       "victories          int64\n",
       "ties               int64\n",
       "defeats            int64\n",
       "goals              int64\n",
       "yellowCards        int64\n",
       "yellowReds         int64\n",
       "redCards           int64\n",
       "photoID           object\n",
       "rater1           float64\n",
       "rater2           float64\n",
       "refNum             int64\n",
       "refCountry         int64\n",
       "Alpha_3           object\n",
       "meanIAT          float64\n",
       "nIAT             float64\n",
       "seIAT            float64\n",
       "meanExp          float64\n",
       "nExp             float64\n",
       "seExp            float64\n",
       "dtype: object"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['playerShort',\n",
       " 'player',\n",
       " 'club',\n",
       " 'leagueCountry',\n",
       " 'birthday',\n",
       " 'height',\n",
       " 'weight',\n",
       " 'position',\n",
       " 'games',\n",
       " 'victories',\n",
       " 'ties',\n",
       " 'defeats',\n",
       " 'goals',\n",
       " 'yellowCards',\n",
       " 'yellowReds',\n",
       " 'redCards',\n",
       " 'photoID',\n",
       " 'rater1',\n",
       " 'rater2',\n",
       " 'refNum',\n",
       " 'refCountry',\n",
       " 'Alpha_3',\n",
       " 'meanIAT',\n",
       " 'nIAT',\n",
       " 'seIAT',\n",
       " 'meanExp',\n",
       " 'nExp',\n",
       " 'seExp']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_columns = df.columns.tolist()\n",
    "all_columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Challenge\n",
    "\n",
    "Before looking below, try to answer some high level questions about the dataset. \n",
    "\n",
    "\n",
    "How do we operationalize the question of referees giving more red cards to dark skinned players?\n",
    "* Counterfactual: if the player were lighter, a ref is more likely to have given a yellow or no card **for the same offense under the same conditions**\n",
    "* Regression: accounting for confounding, darker players have positive coefficient on regression against proportion red/total card\n",
    "\n",
    "Potential issues\n",
    "* How to combine rater1 and rater2? Average them? What if they disagree? Throw it out?\n",
    "* Is data imbalanced, i.e. red cards are very rare?\n",
    "* Is data biased, i.e. players have different amounts of play time? Is this a summary of their whole career?\n",
    "* How do I know I've accounted for all forms of confounding?\n",
    "\n",
    "**First, is there systematic discrimination across all refs?**\n",
    "\n",
    "Exploration/hypotheses:\n",
    "* Distribution of games played\n",
    "* red cards vs games played\n",
    "* Reds per game played vs total cards per game played by skin color\n",
    "* Distribution of # red, # yellow, total cards, and fraction red per game played for all players by avg skin color\n",
    "* How many refs did players encounter?\n",
    "* Do some clubs play more aggresively and get carded more? Or are more reserved and get less?\n",
    "* Does carding vary by leagueCountry?\n",
    "* Do high scorers get more slack (fewer cards) for the same position?\n",
    "* Are there some referees that give more red/yellow cards than others?\n",
    "* how consistent are raters? Check with Cohen's kappa.\n",
    "* how do red cards vary by position? e.g. defenders get more?\n",
    "* Do players with more games get more cards, and is there difference across skin color?\n",
    "* indication of bias depending on refCountry?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Understand how the data's organized\n",
    "\n",
    "The dataset is a single csv where it aggregated every interaction between referee and player into a single row. In other words: Referee A refereed Player B in, say, 10 games, and gave 2 redcards during those 10 games. Then there would be a unique row in the dataset that said: \n",
    "\n",
    "    Referee A, Player B, 2 redcards, ... \n",
    "\n",
    "This has several implications that make this first step to understanding and dealing with this data a bit tricky. First, is that the information about Player B is repeated each time -- meaning if we did a simple average of some metric of we would likely get a misleading result. \n",
    "\n",
    "For example, asking \"what is the average `weight` of the players?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "181.93593798236887"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.height.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "181.93593798236887"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['height'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "tags": [
     "solution"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "181.74372848007872"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(df.groupby('playerShort').height.mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Doing a simple average over the rows will risk double-counting the same player multiple times, for a skewed average. The simple (incorrect) average is ~76.075 kg, but the average weight of the players is ~75.639 kg. There are multiple ways of doing this, but doing a groupby on player makes it so that so each player gets counted exactly once.\n",
    "\n",
    "Not a huge difference in this case but already an illustration of some difficulty."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tidy Data\n",
    "\n",
    "Hadley Wickham's concept of a **tidy dataset** summarized as:\n",
    "\n",
    ">  - Each variable forms a column\n",
    ">  - Each observation forms a row\n",
    ">  - Each type of observational unit forms a table\n",
    "\n",
    "A longer paper describing this can be found in this [pdf](https://www.jstatsoft.org/article/view/v059i10/v59i10.pdf).\n",
    "\n",
    "Having datasets in this form allows for much simpler analyses. So the first step is to try and clean up the dataset into a tidy dataset. \n",
    "\n",
    "The first step that I am going to take is to break up the dataset into the different observational units. By that I'm going to have separate tables (or dataframes) for: \n",
    "\n",
    " - players\n",
    " - clubs\n",
    " - referees\n",
    " - countries\n",
    " - dyads"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Tidy Players Table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "player_index = 'playerShort'\n",
    "player_cols = [#'player', # drop player name, we have unique identifier\n",
    "               'birthday',\n",
    "               'height',\n",
    "               'weight',\n",
    "               'position',\n",
    "               'photoID',\n",
    "               'rater1',\n",
    "               'rater2',\n",
    "              ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Count the unique variables (if we got different weight values, \n",
    "# for example, then we should get more than one unique value in this groupby)\n",
    "all_cols_unique_players = df.groupby('playerShort').agg({col:'nunique' for col in player_cols})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>weight</th>\n",
       "      <th>birthday</th>\n",
       "      <th>rater1</th>\n",
       "      <th>height</th>\n",
       "      <th>position</th>\n",
       "      <th>photoID</th>\n",
       "      <th>rater2</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>playerShort</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>aaron-hughes</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>aaron-hunt</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>aaron-lennon</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>aaron-ramsey</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>abdelhamid-el-kaoutari</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        weight  birthday  rater1  height  position  photoID  \\\n",
       "playerShort                                                                   \n",
       "aaron-hughes                 1         1       1       1         1        1   \n",
       "aaron-hunt                   1         1       1       1         1        1   \n",
       "aaron-lennon                 1         1       1       1         1        1   \n",
       "aaron-ramsey                 1         1       1       1         1        1   \n",
       "abdelhamid-el-kaoutari       1         1       1       1         1        1   \n",
       "\n",
       "                        rater2  \n",
       "playerShort                     \n",
       "aaron-hughes                 1  \n",
       "aaron-hunt                   1  \n",
       "aaron-lennon                 1  \n",
       "aaron-ramsey                 1  \n",
       "abdelhamid-el-kaoutari       1  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_cols_unique_players.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>weight</th>\n",
       "      <th>birthday</th>\n",
       "      <th>rater1</th>\n",
       "      <th>height</th>\n",
       "      <th>position</th>\n",
       "      <th>photoID</th>\n",
       "      <th>rater2</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>playerShort</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [weight, birthday, rater1, height, position, photoID, rater2]\n",
       "Index: []"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# If all values are the same per player then this should be empty (and it is!)\n",
    "all_cols_unique_players[all_cols_unique_players > 1].dropna().head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# A slightly more elegant way to test the uniqueness\n",
    "all_cols_unique_players[all_cols_unique_players > 1].dropna().shape[0] == 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hooray, our data passed our sanity check. Let's create a function to create a table and run thisÂ check for each table that we create."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_subgroup(dataframe, g_index, g_columns):\n",
    "    \"\"\"Helper function that creates a sub-table from the columns and runs a quick uniqueness test.\"\"\"\n",
    "    g = dataframe.groupby(g_index).agg({col:'nunique' for col in g_columns})\n",
    "    if g[g > 1].dropna().shape[0] != 0:\n",
    "        print(\"Warning: you probably assumed this had all unique values but it doesn't.\")\n",
    "    return dataframe.groupby(g_index).agg({col:'max' for col in g_columns})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>weight</th>\n",
       "      <th>birthday</th>\n",
       "      <th>rater1</th>\n",
       "      <th>height</th>\n",
       "      <th>position</th>\n",
       "      <th>photoID</th>\n",
       "      <th>rater2</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>playerShort</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>aaron-hughes</th>\n",
       "      <td>71.0</td>\n",
       "      <td>08.11.1979</td>\n",
       "      <td>0.25</td>\n",
       "      <td>182.0</td>\n",
       "      <td>Center Back</td>\n",
       "      <td>3868.jpg</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>aaron-hunt</th>\n",
       "      <td>73.0</td>\n",
       "      <td>04.09.1986</td>\n",
       "      <td>0.00</td>\n",
       "      <td>183.0</td>\n",
       "      <td>Attacking Midfielder</td>\n",
       "      <td>20136.jpg</td>\n",
       "      <td>0.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>aaron-lennon</th>\n",
       "      <td>63.0</td>\n",
       "      <td>16.04.1987</td>\n",
       "      <td>0.25</td>\n",
       "      <td>165.0</td>\n",
       "      <td>Right Midfielder</td>\n",
       "      <td>13515.jpg</td>\n",
       "      <td>0.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>aaron-ramsey</th>\n",
       "      <td>76.0</td>\n",
       "      <td>26.12.1990</td>\n",
       "      <td>0.00</td>\n",
       "      <td>178.0</td>\n",
       "      <td>Center Midfielder</td>\n",
       "      <td>94953.jpg</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>abdelhamid-el-kaoutari</th>\n",
       "      <td>73.0</td>\n",
       "      <td>17.03.1990</td>\n",
       "      <td>0.25</td>\n",
       "      <td>180.0</td>\n",
       "      <td>Center Back</td>\n",
       "      <td>124913.jpg</td>\n",
       "      <td>0.25</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        weight    birthday  rater1  height  \\\n",
       "playerShort                                                  \n",
       "aaron-hughes              71.0  08.11.1979    0.25   182.0   \n",
       "aaron-hunt                73.0  04.09.1986    0.00   183.0   \n",
       "aaron-lennon              63.0  16.04.1987    0.25   165.0   \n",
       "aaron-ramsey              76.0  26.12.1990    0.00   178.0   \n",
       "abdelhamid-el-kaoutari    73.0  17.03.1990    0.25   180.0   \n",
       "\n",
       "                                    position     photoID  rater2  \n",
       "playerShort                                                       \n",
       "aaron-hughes                     Center Back    3868.jpg    0.00  \n",
       "aaron-hunt              Attacking Midfielder   20136.jpg    0.25  \n",
       "aaron-lennon                Right Midfielder   13515.jpg    0.25  \n",
       "aaron-ramsey               Center Midfielder   94953.jpg    0.00  \n",
       "abdelhamid-el-kaoutari           Center Back  124913.jpg    0.25  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "players = get_subgroup(df, player_index, player_cols)\n",
    "players.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def save_subgroup(dataframe, g_index, subgroup_name, prefix='raw_'):\n",
    "    save_subgroup_filename = \"\".join([prefix, subgroup_name, \".csv.gz\"])\n",
    "    dataframe.to_csv(save_subgroup_filename, compression='gzip', encoding='UTF-8')\n",
    "    test_df = pd.read_csv(save_subgroup_filename, compression='gzip', index_col=g_index, encoding='UTF-8')\n",
    "    # Test that we recover what we send in\n",
    "    if dataframe.equals(test_df):\n",
    "        print(\"Test-passed: we recover the equivalent subgroup dataframe.\")\n",
    "    else:\n",
    "        print(\"Warning -- equivalence test!!! Double-check.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test-passed: we recover the equivalent subgroup dataframe.\n"
     ]
    }
   ],
   "source": [
    "save_subgroup(players, player_index, \"players\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Tidy Clubs Table\n",
    "\n",
    "Create the clubs table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>leagueCountry</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>club</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1. FC NÃ¼rnberg</th>\n",
       "      <td>Germany</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1. FSV Mainz 05</th>\n",
       "      <td>Germany</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1899 Hoffenheim</th>\n",
       "      <td>Germany</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AC Ajaccio</th>\n",
       "      <td>France</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AFC Bournemouth</th>\n",
       "      <td>England</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                leagueCountry\n",
       "club                         \n",
       "1. FC NÃ¼rnberg        Germany\n",
       "1. FSV Mainz 05       Germany\n",
       "1899 Hoffenheim       Germany\n",
       "AC Ajaccio             France\n",
       "AFC Bournemouth       England"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "club_index = 'club'\n",
    "club_cols = ['leagueCountry']\n",
    "clubs = get_subgroup(df, club_index, club_cols)\n",
    "clubs.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "England    48\n",
       "Spain      27\n",
       "France     22\n",
       "Germany    21\n",
       "Name: leagueCountry, dtype: int64"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clubs['leagueCountry'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test-passed: we recover the equivalent subgroup dataframe.\n"
     ]
    }
   ],
   "source": [
    "save_subgroup(clubs, club_index, \"clubs\", )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Tidy Referees Table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>refCountry</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>refNum</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        refCountry\n",
       "refNum            \n",
       "1                1\n",
       "2                2\n",
       "3                3\n",
       "4                4\n",
       "5                5"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "referee_index = 'refNum'\n",
    "referee_cols = ['refCountry']\n",
    "referees = get_subgroup(df, referee_index, referee_cols)\n",
    "referees.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "161"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "referees.refCountry.nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>refCountry</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>refNum</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3143</th>\n",
       "      <td>51</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3144</th>\n",
       "      <td>55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3145</th>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3146</th>\n",
       "      <td>51</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3147</th>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        refCountry\n",
       "refNum            \n",
       "3143            51\n",
       "3144            55\n",
       "3145            21\n",
       "3146            51\n",
       "3147            21"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "referees.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3147, 1)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "referees.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test-passed: we recover the equivalent subgroup dataframe.\n"
     ]
    }
   ],
   "source": [
    "save_subgroup(referees, referee_index, \"referees\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Tidy Countries Table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>nIAT</th>\n",
       "      <th>meanIAT</th>\n",
       "      <th>Alpha_3</th>\n",
       "      <th>nExp</th>\n",
       "      <th>meanExp</th>\n",
       "      <th>seIAT</th>\n",
       "      <th>seExp</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>refCountry</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>712.0</td>\n",
       "      <td>0.326391</td>\n",
       "      <td>GRC</td>\n",
       "      <td>750.0</td>\n",
       "      <td>0.396000</td>\n",
       "      <td>0.000564</td>\n",
       "      <td>0.002696</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>40.0</td>\n",
       "      <td>0.203375</td>\n",
       "      <td>ZMB</td>\n",
       "      <td>49.0</td>\n",
       "      <td>-0.204082</td>\n",
       "      <td>0.010875</td>\n",
       "      <td>0.061504</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1785.0</td>\n",
       "      <td>0.369894</td>\n",
       "      <td>ESP</td>\n",
       "      <td>1897.0</td>\n",
       "      <td>0.588297</td>\n",
       "      <td>0.000229</td>\n",
       "      <td>0.001002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>127.0</td>\n",
       "      <td>0.325185</td>\n",
       "      <td>LUX</td>\n",
       "      <td>130.0</td>\n",
       "      <td>0.538462</td>\n",
       "      <td>0.003297</td>\n",
       "      <td>0.013752</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>19.0</td>\n",
       "      <td>0.167132</td>\n",
       "      <td>TUN</td>\n",
       "      <td>19.0</td>\n",
       "      <td>-0.789474</td>\n",
       "      <td>0.027327</td>\n",
       "      <td>0.111757</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              nIAT   meanIAT Alpha_3    nExp   meanExp     seIAT     seExp\n",
       "refCountry                                                                \n",
       "1            712.0  0.326391     GRC   750.0  0.396000  0.000564  0.002696\n",
       "2             40.0  0.203375     ZMB    49.0 -0.204082  0.010875  0.061504\n",
       "3           1785.0  0.369894     ESP  1897.0  0.588297  0.000229  0.001002\n",
       "4            127.0  0.325185     LUX   130.0  0.538462  0.003297  0.013752\n",
       "5             19.0  0.167132     TUN    19.0 -0.789474  0.027327  0.111757"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "country_index = 'refCountry'\n",
    "country_cols = ['Alpha_3', # rename this name of country\n",
    "                'meanIAT',\n",
    "                'nIAT',\n",
    "                'seIAT',\n",
    "                'meanExp',\n",
    "                'nExp',\n",
    "                'seExp',\n",
    "               ]\n",
    "countries = get_subgroup(df, country_index, country_cols)\n",
    "countries.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>nIAT</th>\n",
       "      <th>meanIAT</th>\n",
       "      <th>countryName</th>\n",
       "      <th>nExp</th>\n",
       "      <th>meanExp</th>\n",
       "      <th>seIAT</th>\n",
       "      <th>seExp</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>refCountry</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>712.0</td>\n",
       "      <td>0.326391</td>\n",
       "      <td>GRC</td>\n",
       "      <td>750.0</td>\n",
       "      <td>0.396000</td>\n",
       "      <td>0.000564</td>\n",
       "      <td>0.002696</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>40.0</td>\n",
       "      <td>0.203375</td>\n",
       "      <td>ZMB</td>\n",
       "      <td>49.0</td>\n",
       "      <td>-0.204082</td>\n",
       "      <td>0.010875</td>\n",
       "      <td>0.061504</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1785.0</td>\n",
       "      <td>0.369894</td>\n",
       "      <td>ESP</td>\n",
       "      <td>1897.0</td>\n",
       "      <td>0.588297</td>\n",
       "      <td>0.000229</td>\n",
       "      <td>0.001002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>127.0</td>\n",
       "      <td>0.325185</td>\n",
       "      <td>LUX</td>\n",
       "      <td>130.0</td>\n",
       "      <td>0.538462</td>\n",
       "      <td>0.003297</td>\n",
       "      <td>0.013752</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>19.0</td>\n",
       "      <td>0.167132</td>\n",
       "      <td>TUN</td>\n",
       "      <td>19.0</td>\n",
       "      <td>-0.789474</td>\n",
       "      <td>0.027327</td>\n",
       "      <td>0.111757</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              nIAT   meanIAT countryName    nExp   meanExp     seIAT     seExp\n",
       "refCountry                                                                    \n",
       "1            712.0  0.326391         GRC   750.0  0.396000  0.000564  0.002696\n",
       "2             40.0  0.203375         ZMB    49.0 -0.204082  0.010875  0.061504\n",
       "3           1785.0  0.369894         ESP  1897.0  0.588297  0.000229  0.001002\n",
       "4            127.0  0.325185         LUX   130.0  0.538462  0.003297  0.013752\n",
       "5             19.0  0.167132         TUN    19.0 -0.789474  0.027327  0.111757"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rename_columns = {'Alpha_3':'countryName', }\n",
    "countries = countries.rename(columns=rename_columns)\n",
    "countries.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(161, 7)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "countries.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Warning -- equivalence test!!! Double-check.\n"
     ]
    }
   ],
   "source": [
    "save_subgroup(countries, country_index, \"countries\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Ok testing this out: \n",
    "test_df = pd.read_csv(\"raw_countries.csv.gz\", compression='gzip', index_col=country_index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "nIAT                  127\n",
      "meanIAT          0.325185\n",
      "countryName           LUX\n",
      "nExp                  130\n",
      "meanExp          0.538462\n",
      "seIAT          0.00329681\n",
      "seExp           0.0137522\n",
      "Name: 4, dtype: object\n",
      "\n",
      "nIAT                  127\n",
      "meanIAT          0.325185\n",
      "countryName           LUX\n",
      "nExp                  130\n",
      "meanExp          0.538462\n",
      "seIAT          0.00329681\n",
      "seExp           0.0137522\n",
      "Name: 4, dtype: object\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for (_, row1), (_, row2) in zip(test_df.iterrows(), countries.iterrows()):\n",
    "    if not row1.equals(row2):\n",
    "        print(row1)\n",
    "        print()\n",
    "        print(row2)\n",
    "        print()\n",
    "        break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "nIAT            True\n",
       "meanIAT         True\n",
       "countryName     True\n",
       "nExp            True\n",
       "meanExp         True\n",
       "seIAT          False\n",
       "seExp           True\n",
       "Name: 4, dtype: bool"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "row1.eq(row2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4.336808689942018e-19"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "row1.seIAT - row2.seIAT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "nIAT           float64\n",
       "meanIAT        float64\n",
       "countryName     object\n",
       "nExp           float64\n",
       "meanExp        float64\n",
       "seIAT          float64\n",
       "seExp          float64\n",
       "dtype: object"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "countries.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "nIAT           float64\n",
       "meanIAT        float64\n",
       "countryName     object\n",
       "nExp           float64\n",
       "meanExp        float64\n",
       "seIAT          float64\n",
       "seExp          float64\n",
       "dtype: object"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>nIAT</th>\n",
       "      <th>meanIAT</th>\n",
       "      <th>countryName</th>\n",
       "      <th>nExp</th>\n",
       "      <th>meanExp</th>\n",
       "      <th>seIAT</th>\n",
       "      <th>seExp</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>refCountry</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>712.0</td>\n",
       "      <td>0.326391</td>\n",
       "      <td>GRC</td>\n",
       "      <td>750.0</td>\n",
       "      <td>0.396000</td>\n",
       "      <td>0.000564</td>\n",
       "      <td>0.002696</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>40.0</td>\n",
       "      <td>0.203375</td>\n",
       "      <td>ZMB</td>\n",
       "      <td>49.0</td>\n",
       "      <td>-0.204082</td>\n",
       "      <td>0.010875</td>\n",
       "      <td>0.061504</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1785.0</td>\n",
       "      <td>0.369894</td>\n",
       "      <td>ESP</td>\n",
       "      <td>1897.0</td>\n",
       "      <td>0.588297</td>\n",
       "      <td>0.000229</td>\n",
       "      <td>0.001002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>127.0</td>\n",
       "      <td>0.325185</td>\n",
       "      <td>LUX</td>\n",
       "      <td>130.0</td>\n",
       "      <td>0.538462</td>\n",
       "      <td>0.003297</td>\n",
       "      <td>0.013752</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>19.0</td>\n",
       "      <td>0.167132</td>\n",
       "      <td>TUN</td>\n",
       "      <td>19.0</td>\n",
       "      <td>-0.789474</td>\n",
       "      <td>0.027327</td>\n",
       "      <td>0.111757</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              nIAT   meanIAT countryName    nExp   meanExp     seIAT     seExp\n",
       "refCountry                                                                    \n",
       "1            712.0  0.326391         GRC   750.0  0.396000  0.000564  0.002696\n",
       "2             40.0  0.203375         ZMB    49.0 -0.204082  0.010875  0.061504\n",
       "3           1785.0  0.369894         ESP  1897.0  0.588297  0.000229  0.001002\n",
       "4            127.0  0.325185         LUX   130.0  0.538462  0.003297  0.013752\n",
       "5             19.0  0.167132         TUN    19.0 -0.789474  0.027327  0.111757"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "countries.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>nIAT</th>\n",
       "      <th>meanIAT</th>\n",
       "      <th>countryName</th>\n",
       "      <th>nExp</th>\n",
       "      <th>meanExp</th>\n",
       "      <th>seIAT</th>\n",
       "      <th>seExp</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>refCountry</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>712.0</td>\n",
       "      <td>0.326391</td>\n",
       "      <td>GRC</td>\n",
       "      <td>750.0</td>\n",
       "      <td>0.396000</td>\n",
       "      <td>0.000564</td>\n",
       "      <td>0.002696</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>40.0</td>\n",
       "      <td>0.203375</td>\n",
       "      <td>ZMB</td>\n",
       "      <td>49.0</td>\n",
       "      <td>-0.204082</td>\n",
       "      <td>0.010875</td>\n",
       "      <td>0.061504</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1785.0</td>\n",
       "      <td>0.369894</td>\n",
       "      <td>ESP</td>\n",
       "      <td>1897.0</td>\n",
       "      <td>0.588297</td>\n",
       "      <td>0.000229</td>\n",
       "      <td>0.001002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>127.0</td>\n",
       "      <td>0.325185</td>\n",
       "      <td>LUX</td>\n",
       "      <td>130.0</td>\n",
       "      <td>0.538462</td>\n",
       "      <td>0.003297</td>\n",
       "      <td>0.013752</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>19.0</td>\n",
       "      <td>0.167132</td>\n",
       "      <td>TUN</td>\n",
       "      <td>19.0</td>\n",
       "      <td>-0.789474</td>\n",
       "      <td>0.027327</td>\n",
       "      <td>0.111757</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              nIAT   meanIAT countryName    nExp   meanExp     seIAT     seExp\n",
       "refCountry                                                                    \n",
       "1            712.0  0.326391         GRC   750.0  0.396000  0.000564  0.002696\n",
       "2             40.0  0.203375         ZMB    49.0 -0.204082  0.010875  0.061504\n",
       "3           1785.0  0.369894         ESP  1897.0  0.588297  0.000229  0.001002\n",
       "4            127.0  0.325185         LUX   130.0  0.538462  0.003297  0.013752\n",
       "5             19.0  0.167132         TUN    19.0 -0.789474  0.027327  0.111757"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like precision error, so I'm not concerned. All other sanity checks pass."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>nIAT</th>\n",
       "      <th>meanIAT</th>\n",
       "      <th>countryName</th>\n",
       "      <th>nExp</th>\n",
       "      <th>meanExp</th>\n",
       "      <th>seIAT</th>\n",
       "      <th>seExp</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>refCountry</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>157</th>\n",
       "      <td>27.0</td>\n",
       "      <td>0.179002</td>\n",
       "      <td>MOZ</td>\n",
       "      <td>31.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.015172</td>\n",
       "      <td>0.032258</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>158</th>\n",
       "      <td>402.0</td>\n",
       "      <td>0.344074</td>\n",
       "      <td>DOM</td>\n",
       "      <td>373.0</td>\n",
       "      <td>0.297587</td>\n",
       "      <td>0.001123</td>\n",
       "      <td>0.005557</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>159</th>\n",
       "      <td>23.0</td>\n",
       "      <td>0.318392</td>\n",
       "      <td>CAF</td>\n",
       "      <td>23.0</td>\n",
       "      <td>-0.130435</td>\n",
       "      <td>0.019218</td>\n",
       "      <td>0.147328</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>160</th>\n",
       "      <td>90.0</td>\n",
       "      <td>0.349965</td>\n",
       "      <td>NPL</td>\n",
       "      <td>101.0</td>\n",
       "      <td>0.366337</td>\n",
       "      <td>0.004713</td>\n",
       "      <td>0.015700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>161</th>\n",
       "      <td>313.0</td>\n",
       "      <td>0.304238</td>\n",
       "      <td>BGD</td>\n",
       "      <td>316.0</td>\n",
       "      <td>0.585443</td>\n",
       "      <td>0.001560</td>\n",
       "      <td>0.012881</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             nIAT   meanIAT countryName   nExp   meanExp     seIAT     seExp\n",
       "refCountry                                                                  \n",
       "157          27.0  0.179002         MOZ   31.0  0.000000  0.015172  0.032258\n",
       "158         402.0  0.344074         DOM  373.0  0.297587  0.001123  0.005557\n",
       "159          23.0  0.318392         CAF   23.0 -0.130435  0.019218  0.147328\n",
       "160          90.0  0.349965         NPL  101.0  0.366337  0.004713  0.015700\n",
       "161         313.0  0.304238         BGD  316.0  0.585443  0.001560  0.012881"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "countries.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>nIAT</th>\n",
       "      <th>meanIAT</th>\n",
       "      <th>countryName</th>\n",
       "      <th>nExp</th>\n",
       "      <th>meanExp</th>\n",
       "      <th>seIAT</th>\n",
       "      <th>seExp</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>refCountry</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>157</th>\n",
       "      <td>27.0</td>\n",
       "      <td>0.179002</td>\n",
       "      <td>MOZ</td>\n",
       "      <td>31.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.015172</td>\n",
       "      <td>0.032258</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>158</th>\n",
       "      <td>402.0</td>\n",
       "      <td>0.344074</td>\n",
       "      <td>DOM</td>\n",
       "      <td>373.0</td>\n",
       "      <td>0.297587</td>\n",
       "      <td>0.001123</td>\n",
       "      <td>0.005557</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>159</th>\n",
       "      <td>23.0</td>\n",
       "      <td>0.318392</td>\n",
       "      <td>CAF</td>\n",
       "      <td>23.0</td>\n",
       "      <td>-0.130435</td>\n",
       "      <td>0.019218</td>\n",
       "      <td>0.147328</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>160</th>\n",
       "      <td>90.0</td>\n",
       "      <td>0.349965</td>\n",
       "      <td>NPL</td>\n",
       "      <td>101.0</td>\n",
       "      <td>0.366337</td>\n",
       "      <td>0.004713</td>\n",
       "      <td>0.015700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>161</th>\n",
       "      <td>313.0</td>\n",
       "      <td>0.304238</td>\n",
       "      <td>BGD</td>\n",
       "      <td>316.0</td>\n",
       "      <td>0.585443</td>\n",
       "      <td>0.001560</td>\n",
       "      <td>0.012881</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             nIAT   meanIAT countryName   nExp   meanExp     seIAT     seExp\n",
       "refCountry                                                                  \n",
       "157          27.0  0.179002         MOZ   31.0  0.000000  0.015172  0.032258\n",
       "158         402.0  0.344074         DOM  373.0  0.297587  0.001123  0.005557\n",
       "159          23.0  0.318392         CAF   23.0 -0.130435  0.019218  0.147328\n",
       "160          90.0  0.349965         NPL  101.0  0.366337  0.004713  0.015700\n",
       "161         313.0  0.304238         BGD  316.0  0.585443  0.001560  0.012881"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create separate (not yet Tidy) Dyads Table\n",
    "\n",
    "This is one of the more complex tables to reason about -- so we'll save it for a bit later. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dyad_index = ['refNum', 'playerShort']\n",
    "dyad_cols = ['games',\n",
    "             'victories',\n",
    "             'ties',\n",
    "             'defeats',\n",
    "             'goals',\n",
    "             'yellowCards',\n",
    "             'yellowReds',\n",
    "             'redCards',\n",
    "            ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dyads = get_subgroup(df, g_index=dyad_index, g_columns=dyad_cols)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>yellowCards</th>\n",
       "      <th>yellowReds</th>\n",
       "      <th>victories</th>\n",
       "      <th>ties</th>\n",
       "      <th>games</th>\n",
       "      <th>defeats</th>\n",
       "      <th>goals</th>\n",
       "      <th>redCards</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>refNum</th>\n",
       "      <th>playerShort</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <th>lucas-wilchez</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <th>john-utaka</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">3</th>\n",
       "      <th>abdon-prats</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pablo-mari</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ruben-pena</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">4</th>\n",
       "      <th>aaron-hughes</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>aleksandar-kolarov</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>alexander-tettey</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>anders-lindegaard</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>andreas-beck</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                           yellowCards  yellowReds  victories  ties  games  \\\n",
       "refNum playerShort                                                           \n",
       "1      lucas-wilchez                 0           0          0     0      1   \n",
       "2      john-utaka                    1           0          0     0      1   \n",
       "3      abdon-prats                   1           0          0     1      1   \n",
       "       pablo-mari                    0           0          1     0      1   \n",
       "       ruben-pena                    0           0          1     0      1   \n",
       "4      aaron-hughes                  0           0          0     0      1   \n",
       "       aleksandar-kolarov            0           0          1     0      1   \n",
       "       alexander-tettey              0           0          0     0      1   \n",
       "       anders-lindegaard             0           0          0     1      1   \n",
       "       andreas-beck                  0           0          1     0      1   \n",
       "\n",
       "                           defeats  goals  redCards  \n",
       "refNum playerShort                                   \n",
       "1      lucas-wilchez             1      0         0  \n",
       "2      john-utaka                1      0         0  \n",
       "3      abdon-prats               0      0         0  \n",
       "       pablo-mari                0      0         0  \n",
       "       ruben-pena                0      0         0  \n",
       "4      aaron-hughes              1      0         0  \n",
       "       aleksandar-kolarov        0      0         0  \n",
       "       alexander-tettey          1      0         0  \n",
       "       anders-lindegaard         0      0         0  \n",
       "       andreas-beck              0      0         0  "
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dyads.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(146028, 8)"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dyads.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>yellowCards</th>\n",
       "      <th>yellowReds</th>\n",
       "      <th>victories</th>\n",
       "      <th>ties</th>\n",
       "      <th>games</th>\n",
       "      <th>defeats</th>\n",
       "      <th>goals</th>\n",
       "      <th>redCards</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>refNum</th>\n",
       "      <th>playerShort</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>140</th>\n",
       "      <th>bodipo</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>367</th>\n",
       "      <th>antonio-lopez_2</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">432</th>\n",
       "      <th>javi-martinez</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>14</td>\n",
       "      <td>7</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>jonas</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>487</th>\n",
       "      <th>phil-jagielka</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>7</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>586</th>\n",
       "      <th>cyril-jeunechamp</th>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>14</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>804</th>\n",
       "      <th>sergio-ramos</th>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>18</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>985</th>\n",
       "      <th>aly-cissokho</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>9</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1114</th>\n",
       "      <th>eugen-polanski</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1214</th>\n",
       "      <th>emmanuel-adebayor</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>7</td>\n",
       "      <td>23</td>\n",
       "      <td>7</td>\n",
       "      <td>10</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          yellowCards  yellowReds  victories  ties  games  \\\n",
       "refNum playerShort                                                          \n",
       "140    bodipo                       0           0          2     1      6   \n",
       "367    antonio-lopez_2              2           0          5     2      8   \n",
       "432    javi-martinez                2           0          4     3     14   \n",
       "       jonas                        0           0          1     4      9   \n",
       "487    phil-jagielka                0           0          2     1      7   \n",
       "586    cyril-jeunechamp             6           0          8     0     14   \n",
       "804    sergio-ramos                 6           1         12     1     18   \n",
       "985    aly-cissokho                 1           0          1     5      9   \n",
       "1114   eugen-polanski               0           0          4     0      8   \n",
       "1214   emmanuel-adebayor            4           1          9     7     23   \n",
       "\n",
       "                          defeats  goals  redCards  \n",
       "refNum playerShort                                  \n",
       "140    bodipo                   3      1         2  \n",
       "367    antonio-lopez_2          1      0         2  \n",
       "432    javi-martinez            7      2         2  \n",
       "       jonas                    4      1         2  \n",
       "487    phil-jagielka            4      1         2  \n",
       "586    cyril-jeunechamp         6      0         2  \n",
       "804    sergio-ramos             5      4         2  \n",
       "985    aly-cissokho             3      1         2  \n",
       "1114   eugen-polanski           4      0         2  \n",
       "1214   emmanuel-adebayor        7     10         2  "
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dyads[dyads.redCards > 1].head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test-passed: we recover the equivalent subgroup dataframe.\n"
     ]
    }
   ],
   "source": [
    "save_subgroup(dyads, dyad_index, \"dyads\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dyads.redCards.max()"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  },
  "toc": {
   "colors": {
    "hover_highlight": "#DAA520",
    "running_highlight": "#FF0000",
    "selected_highlight": "#FFD700"
   },
   "moveMenuLeft": true,
   "nav_menu": {
    "height": "318px",
    "width": "252px"
   },
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 4,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false,
   "widenNotebook": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
